Data Heterogeneity Differential Privacy: From Theory to Algorithm

Traditionally, the random noise is equally injected when training with different data instances in field of differential privacy (DP). In this paper, we first give sharper excess risk bounds DP stochastic gradient descent (SGD) method. Considering most previous methods are under convex conditions, use Polyak-Łojasiewicz condition to relax it paper. Then, after observing that affect machine learning model extent, consider heterogeneity and attempt improve performance DP-SGD from a new perspective. Specifically, by introducing influence function (IF), quantitatively measure contributions various on final model. If contribution made single instance so little attackers cannot infer anything model, do not add it. Based observation, design ‘Performance Improving’ algorithm: PIDP-SGD. Theoretical experimental results show our proposed PIDP-SGD improves significantly.